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Abstract. The formal verification of properties of Hidden Markov Mod¬ 
els (HMMs) is highly desirable for gaining confidence in the correctness 
of the model and the corresponding system. A significant step towards 
HMM verification was the development by Zhang et al. of a family of 
logics for verifying HMMs, called POCTL*, and its model checking algo¬ 
rithm. As far as we know, the verification tool we present here is the first 
one based on Zhang et al.’s approach. As an example of its effective appli¬ 
cation, we verify properties of a handover task in the context of human- 
robot interaction. Our tool was implemented in Haskell, and the ex¬ 
perimental evaluation was performed using the humanoid robot Bert2. 

1 Introduction 

A Hidden Markov Model (HMM) is an extension of a Discrete Time Markov 
Chain (DTMC) where the states of the model are hidden but the observations 
are visible. Typically, an HMM is studied with respect to the three basic problems 
examined by Rabiner in [^. However, to the best of our knowledge, no practical 
model checker exists for HMMs despite their broad range of applications, e.g., 
speech recognition, DNA sequence analysis, text recognition and robot control. 
We describe in this paper a tool for verifying HMM properties written in the 
Probabilistic Observation Computational Tree Logic* (POCTL* [H]), and use 
this tool for verifying properties of a robot-to-human handover interaction. 

POCTL* is a specification language for HMM properties. It is a probabilistic 
version of CTL* where a set of observations is attached to the next operator. 
Zhang et al. m sketched two model checking algorithms for POCTL*, an “au¬ 
tomaton based” approach, and a “direct” approach. We opted for the direct 

* The final version of this paper was accepted in the 13th Interna¬ 
tional Symposium on Automated Technology for Verification and Anal¬ 
ysis (ATVA 2015). The final publication is available at Springer via 
http: //dx.doi.org/10.1007/978-3-319-24953-7_14 
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approach for its lower time complexity. Noticeably, this approach produces a 
DTMC V and a Linear Temporal Logic (LTL) formula (j), so the PRISM [6] 
model checker could be used to verify this property. Such a model checker fol¬ 
lows the automata based approach whose complexity is doubly exponential in 
1^1 and polynomial in \'D\, whereas we implemented the direct method by Cour- 
coubetis et al. [1] whose complexity is singly exponential in \(j)\ and polynomial 
in |X>|, which is also the final complexity of our tool. This direct method repeat¬ 
edly constructs a DTMC and rewrites an LTL formula, such that one temporal 
operator is removed each time while preserving the probability of satisfaction. 

We have named our model checker Marimba. A marimba is a xylophone-like 
musical instrument that is popular in south-east Mexico and Central America. 
Marimba [5] was implemented in Haskell and compiled with GHCi. Our tool 
is available for download from https://github.com/nohernan/Marimba, 

2 Tool architecture and implementation 

Haskell was chosen to code this first version of Marimba since it allows us 
to work in a high-level abstract layer, by providing useful mechanisms like lazy 
evaluation and a pure functional paradigm. Furthermore, Haskell manages 
recursion efficiently; this is a valuable aspect because recursive calls are made 
continuously throughout the execution. As a future work, we consider coding 
Marimba in a language like Java and make it a symbolic model checker. 

Marimba features a command-line interface. Furthermore, instead of work¬ 
ing with a command window, a more user friendly and preferable execution is 
accomplished through the Emacs text editor extended with the Haskell-mode. 

2.1 Marimba’s input and modules 

The hrst input is a .poctl file with the six elements of an HMM T-L, namely 
a finite set of states 5, a state transition probability matrix A, a finite set of 
observations 0, an observation probability matrix B, a function L that maps 
states to sets of atomic propositions from a set APu, and an initial probability 
distribution tt over S. The second input is a POCTL* state formula ^ typed in 
the command window according to the syntactic rules: 

(p ::= true | false | a \ {^(P) \ \ (<? A | 

(/.::=<!> I (-</.) I I \ (Xo<^) | | 

where a G APu, o S 0, n G N, p G [0,1], and [xi G {<, <, >, >}. In addition, we 
define as a shorthand for Voer? provided 17 C 0. We examine below 
the six Haskell modules that constitute Marimba. 

ModelChecker.hs performs the initial computations of the model checker for 
POCTL*. It recursively finds a most nested state subformula of not being 
a propositional variable, and the states of T-L that satisfy it. On the one hand, 
finding the states satisfying a propositional subformula is straightforward. On 
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the other hand, we invoke the module Direct Approach.hs to obtain the states 
satisfying a probabilistic state subformula. Next, this module extends the labels 
of such states with a new atomic proposition a. In the state subformula being 
addressed is replaced by a. The base case occurs when we reach a propositional 
variable, so we return the states that have it in their label. 

Direct Approach.hs transforms the HMM H into a DTMC D, and removes 
from the specification the observation set attached to the next operator X by 
generating a conjunction of the observation-free X with a new propositional 
variable. Thus, we obtain an LTL formula that is passed, together with D, to 
the module Courcoubetis.hs. The new propositional variables are drawn from 
the power set of observations. Remarkably, it is not necessary to compute such 
a power set since the label of a state in V is easily calculated. 

Courcoubetis.hs implements a modified version of the method by Courcou- 
betis et al. to find the probability that an LTL formula is satisfied in a DTMC. 
In this module, when dealing with the 14 and 14-^ operators, we apply ideas 
from [To] for computing a partition of states of D. Moreover, to handle the 14 
operator we have to solve a linear equation system. To that end, we use the 
linearEqSolver library |3], which in turn executes the Z3 theorem prover [2]. 

Lexer.hs and Parser.hs are in charge of the syntactic analysis of the input. 
Finally, Main.hs is loaded to start Marimba. This module manages the interaction 
with the user, and starts the computation by passing control to ModelChecker.hs. 

In a typical execution. Marimba prompts the user to enter a .poctl file path. 
Next, our tool asks whether or not the user wants to take into account the 
initial distribution in the computation of the probability of satisfaction. This 
choice corresponds to opposite ideas presented in [1] and m, i.e., the method 
by Courcoubetis et al. uses the initial distribution to define their probability 
measure, contrary to that defined by Zhang et al. Afterwards, a POCTL* formula 
has to be entered. Marimba returns the list of states satisfying this formula, and 
asks the user whether there are more formulas to be verified on the same model. 

The .poctl file is simply a text file where the elements of an HMM are de¬ 
fined, e.g., the set of states is defined by the reserved word States, and if the 
model consists of five states, we write States=5. Likewise, POCTL* formulas 
have a natural writing, for example, 'P<o.i(X{oj}a) is typed as P [<0.1] (X_{l}a). 

3 Verification of a human-robot interaction 

We applied Marimba to a real-world example, namely the verification of the 
robot-to-human handover task [4] using the robot Bert2 [7] at the Bristol 
Robotics Laboratory (BRL). The robot’s decision to release the object during 
the handover task is determined by an HMM |4]. Figure [T] presents the state 
diagram of the HMM corresponding to the basic handover interaction, where 
the label L{s) is defined for each state. 

Next, we initialise A, B and tt of the HMM as follows. The process starts at 
state Robot not hold, so its initial distribution value tti is almost one, while the 
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L(l) = {rnh} 
State 1 


L(2) = {rpu} 
State 2 


other states have initial distribution values close to zero. The initial matrix A 
must encourage the transitions shown in Figure [T] To initialise B, we consider 
as observations the ordered pairs whose first and 
second components are the index and middle fin¬ 
ger metacarpophalangeal joint motor current val¬ 
ues, respectively. By the Cartesian product of these 
values, we obtain 56,404 observations. Since these 
observations are merged with the states to generate 
the DTMC passed to Courcoubetis.hs, and the size 
of a formula could grow considerably by associating 
the next operator with up to 56,404 observations, 

Marimba’s execution is not practical under these 
circumstances. Vector quantisation [ 8 ] was used to 
reduce the number of observations to just 13, which 
were taken to initialise matrix B. Thus, the initial 
ordered pairs are grouped into 13 regions of the 
plane representing the observations. 

To make reliable estimates, we collected observations from 50 handover ex¬ 
periments on Bert2. These observations were used to train the initial HMM 
with the reestimation method found in the solution of Rabiner’s Problem 3 [9]. 



Fig. 1. The labelled states in¬ 
volved in the basic handover 
process. 


Liveness properties. A liveness property requires that a good thing happens 
during the execution of a system. For example, we would like to know whether 
the model generates the sequence of observations O = oi, 02 , 03,04 where 01,02 S 
{3,4,6} and 03,04 € {3,4,11}, with probability greater than 0.88, that is, 
T’>0.88(X{3,4,6}(X{3_4^6}(X{3,4,ii}(X{3_4^ii}true)))). Interestingly, this property 
is a generalisation of Rabiner’s Problem 1 [^. Marimba’s execution for this 
property is found in Figure [2j The inputs are the trained HMM, defined in 
ModelBert2 .poctl, and the previous formula. The output returned by Marimba 
is State 4. Hence, the model starting at state User grab is likely to generate O. 


Main> main 

Enter the file name where the HMM is located. 
examples/ModelBert2 .poctl 

Would you like to consider each state as if it were the initial 
state, i.e., as if it had initial distribution value equal to 1? y/n: j 
Enter the POCTL* formula we are interested in. 

P[>0.88] (X_{3,4,6}(X_{3,4,6}(X_{3,4,11}(X_{3,4,11}T)))) 

The states that satisfy it are: 

(Probability of satisfaction of each state: [4.998198505964186e-10, 

4.08659792160621e-6,7.508994137303159e-3,0.8915357419467848]) 
[4] 

Do you want to continue checking more specifications? y/n: n 
Fig. 2. Verifying a property with Marimba. 

A second liveness property states that with probability at least 0.9, Bert2 
releases the object when the user grabs it. The POCTL* formula for this property 
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is 'P>o. 9 (rh A (rh U (ug A ug W rnh))). Marimba outputs State 3, i.e., the speci¬ 
fication is satisfied when the starting state is Robot hold. So, we expect Bert2 
to hold the object, and let it go when the user grabs it. 

Safety properties. A safety property establishes that a bad thing does not 
occur during the execution of a system. For instance, with probability less than 
0.05, Bert2 abandons its serving position with the user not grabbing the object, 
that is, 'P<o.o 5 (rh A Xe(rnh V rpu)), where O is the set of observations. Our 
model checker returns {1, 2, 3,4} as the set of states satisfying this property. We 
conclude that it is unlikely that the model, being at state Robot hold, reaches a 
state other than User grab, that is. Robot not hold or Robot pick up. 

The satisfaction of the previous three specifications provides us with confi¬ 
dence that Bert2 reliably performs the handover interaction specified above. 

On an Intel® Core"'"^ i3 1.70GHz computer with 4GB in memory. Marimba 
takes 28.55s to compute the states satisfying the first liveness formula. The time 
required for checking the other two properties studied here is around 0.06s. 

Further examples are given in the examples folder and user’s manual that 
come with Marimba’s source code. 

4 Conclusions 

Since the automatic verification of properties of HMMs seems to be an unat¬ 
tended problem, we present here Marimba, a Haskell implementation of the 
model checking algorithm for POGTL* [llj . This model checking algorithm was 
slightly modified to carry out its computations in a real program. Marimba’s 
calculation is basically broken out in three stages that are coded in the modules 
ModelChecker.hs, Direct Approach, hs and Courcoubetis.hs, such that the involved 
components, steps and transformations are well arranged throughout the im¬ 
plementation. Finally, we have successfully applied Marimba to verify relevant 
properties of a handover interaction from the robot Bert2 to a human. 
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IN113013 and Conacyt 221341, and especially thank the BRL staff for their 
assistance operating the robot Bert2. E. Magid and K. Eder have been sup¬ 
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5 Appendix 

Technical details and formal definitions concerning HMMs and the POCTL* 
formalism are presented next. 

5.1 Hidden Markov Model 

An HMM has two layers, one on top of the other. The stochastic process be¬ 
tween states on the underlying layer is hidden, and can be seen only through 
the stochastic process on the external layer that effectively produces a visible 
sequence of observations. 

Definition 1. A labelled Hidden Markov Model [5] consists of a tuple 
H — {S^ A,0, B, L,Tr), where: 

- S = {5'o, Si ,..., ^n-i} is a finite set of states; 

- A is a state transition probability matrix, such that: 

E TL 1 

^ Qij = 1 0 < i < n — 1] 

- 0 = {uo,ui,... ,Vm-i} is a set of m observations; 

- B is the observation probability matrix, B = {bj{k)} with 

bji^) — P[vk I Sj], 0 < j < n — 1, 0 < k < m — 1; 

- L:S ^ maps states to sets of atomic propositions from a set AP-^; 

- t: is an initial probability distribution over S, such that: 

E Th - 1 

TTi = 1. 

i—0 

An execution of the system which is being modelled by an HMM is repre¬ 
sented by a path. 

Definition 2. A path [11] is a sequence (sqjOo), (si,oi), ..., where st € S, ot € 
6 *: flsiSi+i > 0 and bgfioi) > 0, \/i > 0. A path can be finite (uj^'") or infinite (uj). 

We denote the {i -\- l)st state of uj by uJs{i), and the {i l)st observation by 
uJo{i)- The suffix (si,Oi), (s^+i, Oi+i),... of w is denoted by uj[i]. We denote the 
sets of all finite and infinite paths in H, starting with a pair whose state is s, by 
Path^"’^ and Path^, respectively. 

To quantify the probability that an HMM behaves in a certain way, we de¬ 
fine the measure Pr^ over the set Path^. The basic cylinder set induced by 
the cylinder = (sq, oq), (si, oi),..., (sfe, Ofe) is defined as C{uj^'’') = {w G 
Path^ I Vf G {0, ...,fc} {uJs{i) = Si A uJo{i) = Oi)}. Let Us be the small¬ 
est cr-algebra on Path^ which contains all basic cylinder sets C{uj^'"), where 
Lu^'" = (s, oo ),..., (sfc. Ok) G Path^"’^. We define Pr^ on Bg as, 

Pr^ ^C((s, Oo),..., (sk, Ok))'j = TTsbsioo) as,_-,s,bg. (o^). 
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Let E be the smallest cr-algebra on Path^ containing all basic cylinder sets, 
such that Path^ is the set of paths in TL with no constraint on the state of the 
initial pair. In [5], the probability measure Pr-^ on E is defined in terms of Prg. 

We quantify the probability that an HMM behaves in a certain way by iden¬ 
tifying the set of paths that satisfy a formula, and then using Pr-H (or Prg). 


5.2 POCTL* 

The Probabilistic Observation Computational Tree Logic* (POCTL* [11]) has 
a next operator equipped with an observation constraint. 

Definition 3 (Syntarx). Let H = {S, A, O, B, L, tt) be an HMM defined over the 
set of atomie propositions APu. The syntax of POCTL* is defined as follows: 

<P ::= true \ false | a \ (-i^) | V | [<1> f\<T) \ 

4>:-.= d> \ (-</>) I I I (Xo<(>) I I {fiUfi), 

where a G AP-u, o G 0, n G N, p G [0,1], and ixiG {<, <, >, >}. We distinguish 
between state formulas and path formulas (j). 

Definition 4 (Semantics). Let TL = (5, A, 0, B, L, tt) be an HMM. For any 
state s G S, the satisfaction relation ^ is inductively defined as 


s 1= true Vs G S, 
s ^ false Vs G S, 
s 1= a iff a € L(s), 
s 1= iff s\L(l>, 


s \= Fi\/ ‘1>2 iff s ^ 4^1 V s 1= 4^2) 
s ^ A 4*2 iff s 1= 4^1 A s 1= 4^2) 
s h 'P\xp{4') iff Prsjw G Path^ | w |= (/)} C=<l p. 


For any path uj, the satisfaction relation is defined as 

Uj\=<p iff UJs{0) \= OJ \= (j)i\/ (j)2 iff UJ \= (j)i\/OJ \= 4 i2, 

UJ \= iff UJ ^ 4>, UJ \= (j)i A <p2 iff UJ \= <pi A UJ \= 4>2, 

uj\=Xo4> ijf uJo{0) = o A uj[l] \= (j), 

UJ \= ())iW -"())2 iff 3j < n. {uj[j] \= (j) 2 Atfi < j. uj[i] \= fii), 

uj\= fii 11(1)2 iff > 0. {uj[j] 1= (^2 A Vi < j. uj[i] ^ (fi). 

Let f2 C 0^ we write as a shorthand for Therefore, 

UJ \= Xi70 iff a;o(0) G 17 A a;[l] |= (j). 


6 Model checking algorithm 

Let TL = {S, A, 0, B, L, tt) be an HMM, s be a state in S, and he a POCTL* 
state formula. Next, we explain a method to know whether s ^ ^ holds or not. 


Marimba: A Tool for Verifying Properties of Hidden Markov Models 9 


6.1 Stage One 

According to m, the model checking algorithm starts by taking a most deeply 
nested state subformula S' of <P, such that W is not an atomic proposition. It is 
straightforward to find the states in S that satisfy W when it is propositional. 
To obtain the states that satisfy W when it is of the form Poop(</'); stage two of 
the model checker is invoked. Once we determine the states satisfying If", their 
label is extended by a new atomic proposition a^. Next, W is replaced by in 
<P. The algorithm proceeds recursively, unless ^ itself is replaced by a^; in such 
case the algorithm returns states s, with € L{s). 


6.2 Stage Two 

To identify the states that satisfy we follow the direct approach that 

transforms the original % into a DTMC V = (S'®, A®, L®, tt®), where 

- S'^ = Sx0, - L®(s, o) = L{s) U {I? C 0 I o G f?}, 

- -4®((s,o), (s', o')) = ass' ■ bs'io'), - = tTs ■ 6 ^( 0 ), 

that is defined over the set of atomic propositions AP-p = APu U {12 117 C 0}. 
The argument of V, i.e., </>, is modified to obtain (f/ in a way that every occurrence 
oi'K.Qip is replaced by 17AXi^. Notice that 17 is a new atomic proposition defined 
in APp. 


6.3 Stage Three 

As stated in m, stage three recursively constructs a new DTMC 22' by applying 
the transformations Cx, Cu and C^Kn, which are performed for each occurrence 
of X, U and respectively. To show how the transformations work, we focus 
here on 0x- It takes X(^ as an innermost subexpression of 4>'. Then, it partitions 
the states of V into three disjoint subsets, S'® = S^^^ U S'^° U S^, where: 

- consists of the states whose transitions are only into states satisfying (p. 

- S^® consists of the states whose transitions are only into states satisfying ^ip. 

- consists of the states with transitions to both states satisfying p and 
states satisfying -^p. 

Let qu denote the probability that X:^ is satisfied starting from state u G 5'®. 
We know that = 1 if w G and = 0 if m G 5''^°. Otherwise, qu = 

^„A®(m,w), where the sum ranges over all successor states u of m satisfying 
formula p. Let qH = 1 — qu- Moreover, the new DTMC V is defined over the set 
APpi = APp U {^}, where ^ is a new atomic proposition representing 'X.p. 

States of T>'. For each u G 5”^^^ there is a new state (u, ^) in V . For each u G 
5''^° there is a new state (m, -i^). And for each u £ S'^ , there are two new states 
(u,5) and (it,- 1 ^). We define L® (it,^) = LP{u) U j^} and L® (it,-i^) = L^{u). 
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Transitions of The transition probability of —>• {v,^ 2 ), with G 

{C; and i G {1,2}, is defined as being equal to the probability that T>, being 
at state u, transitions next to state v, and starting from state v onward satisfies 
property ^ 2 , conditioned on the event that in state u it satisfies property 

Initial distribution of X>'. If u S U 5''^°, then ttF , , = liP. If u G 5'’, 
then there are two states in T)' for u, namely (u,^) and (u,-i^), with initial 
probabilities tt® • and ttJ’ ■ q^, respectively. Furthermore, ip is obtained by 
replacing Xtp by ^ in (/)'. 

If (f)' originally has k temporal operators, the algorithm applies k times the 
appropriate transformations Cx, Cu and Cif<n, to finally get the DTMC 
and the propositional formula It is proved in that s ^ 'Pt^p{4i) iff 

oe© \ '---' / 




Since is a propositional formula, Pr^ojcr ^ Path^ | a |= il)^} is . 



